(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



CORRECTED VERSION 



(19) World Intellectual Property Organization 
International Bureau 

(43) International Publication Date 
13 July 2000 (13.07.2000) 




PCT 



(10) International Publication Number 

WO 00/41110 Al 



(51) International Patent Classification 7 : G06F 17/60 

(21) International Application Number: PCT/US99/29432 

(22) International Filing Date: 

10 December 1999 (10.12.1999) 



(25) Filing Language: 

(26) Publication Language: 
(30) Priority Data: 



English 
English 



09/223,426 



30 December 1998 (30.12. 1998) US 



(71) Applicant and 

(72) Inventor: D'ALESSANDRO, Alex [US/US]; 1925 Seag- 
ull Drive, Clearwater, FL 33764 (US). 

(74) Agent: HOPEN, Anton, J.; Smith & Hopen, P.A., 15950 
Bay Vista Drive, Clearwater, FL 33760 (US). 



(81) Designated States (national): CA, JP, ZA. 

(84) Designated States (regional): European patent (AT, BE, 
CH, CY, DE, DK, ES, H, FR, GB, GR, IE, IT, LU, MC, 
NL, PT, SE). 

Published: 

— With international search report. 

(48) Date of publication of this corrected version: 

7 December 2000 

(15) Information about Correction: 

see PCT Gazette No. 49/2000 of 7 December 2000, Section 

n 

For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations'' appearing at the begin- 
ning of each regular issue oftlie PCT Gazette. 



(54) Title: SURVEY SYSTEM TO QUANTIFY VARIOUS CRITERIA RELATING TO THE OPERATION OF AN ORGANIZA- 
TION 



10 



30- 
40- 

50- 



Support Staff 



i f 80 i 



130 



(57) Abstract: A system for providing accurate, quantifiable and reproducible assessments of an organization's performance based 
^j. on predetermined criteria. The system includes a telecommunications infrastructure administered by a survey administrator (20). 
**** Employees or non-employees of a company or other organization (10) to be evaluated log onto the system and answer a plurality 
2 of questions relating to various aspects of the business entity's operations. The information may be obtained through Internet com- 
munications (70) t through touch-tone telephone systems (300), and in many other ways including the use of personal computers, 
diskettes and email, or even manual means such as pencil and paper where penciled-in answers are read by a scanner. The data is 
^ stored in a database (100) and subsequently analyzed by the survey administrator (20) for evaluation and forecasting of the business 
^ entity's performance. 
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SURVEY SYSTEM TO QUANTIFY VARIOUS CRITERIA RELATING TO THE OPERATION OF AN ORGANIZATION 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

5 The present invention relates generally to a survey system and apparatus for 

evaluating the results. More particularly, it relates to a remotely accessible system for the 
collection of employee or non-employee survey responses to quantify various criteria 
relating to the operation of an organization. 

2. Description of the Prior Art 

10 In an increasingly competitive marketplace, business entities strive to improve 

leadership, quality, customer satisfaction and other criteria that directly or indirectly relate 
to the ultimate profitability of the enterprise. Empirical evidence has established that 
business operations that excel in distinct and measurable attributes are far more likely to be 
profitable in commerce. As an example, the "Baldrige Index" is made up of publicly traded 

15 U.S. companies that have received the Malcolm Baldrige National Quality Award during 
the years 1988 to 1996. The Secretary of Commerce and the National Institute of Standards 
and Technology ("NIST") were given the responsibility, under Public Law 100-107, to 
develop and administer the Award with cooperation and financial support from the private 
sector. NIST "invested" a hypothetical $1,000 in each of the six whole company winners of 

20 the Baldrige Award. The investments were tracked from the first business day of the month 
following the announcement of award recipients (or the date they began public trading) to 
Dec. 1, 1997. Adjustments were made for stock splits. Another $1,000 was hypothetical!} 
invested in the Standard and Poor's Index 500 ("S&P") at the same time. NIST found that 
the group of six outperformed the S&P 500 by more than 2.7 to 1, achieving a 394.5 

25 percent return on investment compared to a 146.9 percent return for the S&P 500. 

Therefore, it is desirable for an organization to administer periodic assessments of its 
operations. 

The detailed assessment of a business operation is often a difficult, expensive, and 
time-consuming task. Typically, upper level management may knowingly or inadvertently 
30 affect the accurate measure of information gathered for the assessment. Outside consultants 
are often employed to interview and observe the operation of the enterprise on-site. 
However, management may wish to influence the data for a number of reasons. Upper 
level management may be greatly affected by the results of the evaluation and may attempt 
to direct the outside consultant only to well performing operations, shielding problem areas 
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from discovery. In addition, the outside consultant may interview subordinate employees 
in the presence of their supervisor. This creates a poor environment for gaining candor 
from the subordinate employee on potential areas of improvement that are the 
responsibility of the supervisor. 
5 An evaluation by an outside consultant may require the consultant to travel to 

different geographical locations to assess a large operation with continuity. The consultant 
may also require the employee to stop productive work to allow time for the assessment 
interview. It is time-consuming for an independent consultant to conduct individual 
employee interviews, record the data, and assemble the information into a useful form. 
10 Furthermore, should a business wish to conduct periodic evaluations, there is no guarantee 
that the same consultant will be available. Therefore, the company cannot be assured that 
the next independent assessment performed will have the same consistency. Nor is there 
any assurance that secondary evaluations will produce meaningful results comparable to 
previous exercises. 

15 Should the company attempt to apply known economic principles to an "in-house" 

self-assessment, there is the potential that more harm than good may come from the 
endeavor. Many organizations begin the process of self-assessment with a shallow 
understanding of the performance criteria sought or the optimal method in which to gather 
the information. This self-assessment suffers from an inseparable relationship between the 

20 company's own evaluator and the upper management that may exert influence, not only on 
the examinees, but also on the examiner. 

Previous attempts have been made to provide business-related surveys such as 
described in U.S. Patent No. 5,551,880 to Bonnstetter et al (the '880 patent) which is 
incorporated herein by reference. The '880 patent describes a system for predicting the 

25 potential success of an individual for a particular job or task. A survey is conducted 

wherein the employee or potential employee submits information on behavioral and value 
preferences. The information is then analyzed and compared against standards for behavior 
and values previously resolved for specific employment In a preferred embodiment of the 
'880 patent, the behavior and value questions are administered through software and 

30 evaluated via a predetermined algorithm. However, the system analyzes the individual 
employee and not the business entity as a whole. Nor does the '880 patent describe an 
independent party to administer the survey to ensure confidentially, honesty, and a full 
disclosure of the employee's perceptions of the business entity. 
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Therefore, there is a need in the art for a method and apparatus to provide a reliable 
self-assessment survey process that can be easily administered and scored with the 
accuracy and completeness of a well developed written narrative self-assessment. 

There is a further need in the art to integrate existing telecommunication 
5 technologies to avoid the time and labor intensive ordeal of the paper and pencil process or 
other traditional means of survey administration. This would allow the database of both 
individual responses and cumulative data to be completely external to the organization 
being assessed. 

There is a further need in the art to provide confidentiality to organizations and 
10 their employees participating in the survey. This provides a distinct opportunity to 

benchmark the data to industry peers and to provide the objective assurance that all data is 
accurate and verifiable by a reliable survey administrator organization. 

However, in view of the prior art in at the time the present invention was made, it 
was not obvious to those of ordinary skill in the pertinent art how the identified needs 
15 could be fulfilled. 



SUMMARY OF THE INVENTION 
The present invention solves significant problems in the art by providing a 
telecommunications infrastructure that supports the efficient, confidential and accurate 
20 measure of performance criteria relating to the performance of an organization. A 
predetermined set of performance criteria are measured by the use of an automated 
employee and non-employee interview system whereby recorded responses are stored in a 
database which is utilized for a quantitative evaluation of various aspects of a business 
enterprise. 

25 The present invention comprises an array of survey questions for evaluating a 

plurality of predetermined criteria relating to the performance of an organization. A survey 
administrator controls a data-gathering interface means for employees or non-employees of 
the business to confidentially record answers to the survey questions. For each individual 
employee survey, the data-gathering interface records the employee's identification, their 

30 job classification and a business identifier. Pertinent data is also collected for each non- 
employee participating in the survey. The data-gathering interface may comprise an 
Internet web server, a touch-tone telephone entry system, a paper and pencil system that 
compiles the answers by scanning them automatically, and other existing or to-be- 
developed interface means as well. For example, an additional data-gathering interface 
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means could be provided in the form of a personal computer having appropriate software. 
A respondent could save his or her survey responses on a diskette and mail the diskette to 
the survey administrator, or email the responses to the survey administrator as an attached 
file. This would enable a respondent to prepare answers and send them to the survey 
5 administrator while the respondent is on a flight, on the road, or otherwise away from a 
permanent office environment. 

The survey administrator maintains a database of the responses to the survey 
questions and has the ability to archive past response data. The survey administrator then 
applies a selected formula to the survey data to evaluate the performance of the 

10 organization and produces a quantifiable result based on a plurality of business categories. 

In a preferred embodiment, an organization wishing to evaluate the performance of 
various key aspects of its operations selects one or more business performance models. 
Each business performance model is measured using predetermined questions for 
employees of the business. A survey administrator performs the administration of the 

1 5 predetermined questions so that confidentially in the interview process insures candid 
answers to the questions. In one embodiment, responses to the questions are gathered 
utilizing an Internet web server and are stored either locally on the server or at a remote 
location for the survey administrator to later compile and analyze. The web server provides 
the opportunity to quickly and efficiently gather information, particularly where various 

20 offices of an organization are separated by substantial geographic distance. The web server 
also may operate 24 hours per day at relatively low cost and permit a plurality of 
employees to enter in responses at the same time. 

In an alternative embodiment, a telephone system utilizing dual-tone multi- 
frequency ("DTMF") input provides a means for employees to confidentially record 

25 information relating to the performance of a company. The DTMF entry has the advantage 
of not requiring the employee to have a machine capable of accessing the web server over 
the Internet. Using DTMF entry, the employee logs on using a code provided by the survey 
administrator. Once within the telephone system, pre-recorded questions are audibly 
played to the employee. At the conclusion of each question, the employee is prompted to 

30 record their response by depressing numerals on the telephone. 

In another embodiment, non-employees participate in the survey as well. However, 
the questions propounded to non-employees may differ from the questions propounded to 
employees. 
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The data is readily compiled into a useable database format. At the conclusion of 
the data gathering stage, the survey administrator can easily analyze the recorded 
information and produce a report to the business entity evaluating the entity based on the 
pre-selected business performance criteria desired. Furthermore, the operation of the 
5 invention is highly reproducible and consistent A baseline assessment can be repeated 
over time, allowing the organization to track quantifiable results and improvements. 

It should be noted that the cunrent invention has a wide range of applications which 
may include heath care accreditation, educational institution assessment, and similar tasks. 
For example, the current invention may be applied to a Performance-based Organizational 
1 0 Effectiveness & Efficiency Tracking Program ("POET') wherein the organizational 
effectiveness and efficiency of child welfare organizations are measured in addition to 
performance indicators essential to success in a managed care environment. 

An advantage of the invention is that a complete assessment of a large corporation 
may be completed more rapidly and efficiently than using independent consultants that 
1 5 interview onsite. The reduced costs and speed permit a business to conduct more frequent 
performance evaluations that can be utilized to operate the business more profitably. 

Another advantage of the invention is that employees of the business are provided 
with the opportunity to submit honest and candid answers to potentially sensitive 
questions. Because the opportunity for external influence to affectthe employee's response 
20 is curtailed, the final evaluation of the business entity is more likely to uncover problem 
areas in its operations that may be subsequently improved. 

Another advantage of the invention is that the infrastructure of the evaluation is 
highly consistent over time. This permits the business entity to periodically measure itself 
against previous assessments. 
25 These and other important objects, advantages, and features of the invention will 

become clear as this description proceeds. 

The invention accordingly comprises the features of construction, combination of 
elements, and arrangement of parts that will be exemplified in the description set forth 
hereinafter and the scope of the invention will be indicated in the claims. 

30 

BRIEF DESCRIPTION OF THE DRAWINGS 
For a fuller understanding of the nature and objects of the invention, reference 
should be made to the following detailed description, taken in connection with the 
accompanying drawings, in which: 
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FIG. 1 is a diagrammatical view of the survey process according to the invention; 

FIG. 2 is a diagrammatical view of an Internet-based method of retrieving and 
recording employee responses; 

FIG. 3 is an illustrative example of a survey question presented to a user from an 
5 Internet web server, 

FIG. 4 is a diagrammatical view of a telephone-based method of retrieving and 
recording employee responses; and 

FIG. 5 is an illustrative example of an organization's performance scorecard based 
on the Malcolm Baldrige criteria. 

10 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
Referring initially to FIG. 1, it will there be seen that the organization subject to 
the survey is denoted by the reference number 10 as a whole. Virtually all business entities 
have the following job classifications: executive staff 30, management staff 40, support 

1 5 staff 50, and customer contact staff 60 as integral employee components which support 
most all business functions of the enterprise. It is each of these job classifications of 
employees who form the internal pool of respondents of the business entities. The login 
procedures for these internal respondents will be discussed in more detail later. 

In another embodiment of the invention, an external pool of respondents is also 

20 included. The external pool of respondents may include consumers, customers or clients of 
the business entity, /.<?., those who are not employees or otherwise a part of the internal 
group of respondents. The external pool of respondents may also be adjusted as desired to 
include members of the board of directors or trustees, suppliers or vendors, people who 
perform outsourced operations, competitors, partners, consultants, or any other relevant 

25 external respondent entity. The login procedure for these external respondents is similar to 
the login procedure for internal respondents, except that said login procedure identifies all 
members of the external pool of respondents as such and by their specific category such as 
customer, vendor, or the like. External respondents may also be queried as to their age, 
frequency of contacts with the business entity, and other information deemed pertinent. 

30 In many organizations there are one or more consultants or holders of other job 

titles who are on the business premises daily and whom might seem to be employees of the 
organization even though they are technically independent contractors. There are other jobs 
as well where the distinction between employee or non-employee may not be easily 
ascertainable. This invention does not depend upon any legal analysis of who should be 
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considered internal or external to the organization as a matter of legal definition because in 
some cases a non-employee such as an outside consultant might best be included as a 
member of the internal pool of respondents for survey purposes. Therefore, for 
convenience purposes, and not in any strict legal sense, the members of the pool of internal 
5 respondents are herein deemed employees of the organization, and the members of the pool 
of external respondents are herein deemed non-employees of the organization. A survey 
administrator is free to determine the respective members of the pools of internal and 
external respondents in any way that best serves the purposes of the survey without regard 
to strict legal determinations of the legal status of the individual members of either pool. 

10 The system obtains information from individual employees or other respondents 

regarding their perception of performance criteria existing in the workplace. This is 
accomplished by utilizing a performance criteria survey. This survey is administered and 
controlled by a survey administrator denoted 20. The survey data is obtained by a data- 
gathering interface 70. The data-gathering interface may comprise a variety of 

15 telecommunication mediums which may include touch-tone telephone systems, hypertext 
language format ("HTML") Internet web servers, client-server wide area networks 
("WANs") or local area networks ("LANs"). Data-gathering interface 70 acts as a conduit 
to receive survey data 90 in response to a plurality of predetermined survey questions. 
Each question contains several important aspects of a business function linked to vital 

20 performance abilities of the organization. 

The results of the survey are stored in a database 100 for a baseline analysis of 
performance criteria and may also be used longitudinally in later intervals of time. Prior 
survey data from prior years 110 and 120 may be archived by the survey administrator 20 
and later utilized as a base-line reference to the current evaluation of the organization. 

25 Once a requisite number (^respondents or all respondents complete surveys, an analysis 
140 is performed on the survey data 90 by the survey administrator 20 and then 
communicated back to the organization 10. 

FIG. 2 provides an overview of an Internet-based data and tracking system. A 
remote user terminal 160 accesses an Internet web server 70 through a public switched 

30 network ("PSN") 190. The survey administrator 20 provides the organization 10 with a 
series of codes for the login 210 and test administration to begin. Typically, the 
organization 10 will choose an alphanumeric password often digits or less for 
identification of the organization 10 as a whole. The survey administrator 20 records said 
password and opens it for usage in protected area of the website. Next the survey 
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administrator 20 assigns a chronological range of numbers sufficient to match the total 
number of respondent employees or non-employees elected and eligible to sit for the 
survey. To begin the login 210 process, the respondent accesses the world wide web and 
web server 70 using its uniform resource locator ("URL"). The URL may be provided as a 
5 numerical-based Internet protocol ("IP") address or may be accessed by a registered 
domain name via a domain name server ("DNS"). 

It should also be noted that alternatives to web-based Internet communications may 
be employed. The alternatives may include 3270 emulation, VT-100, ASCII or separate 
client-server applications. 

10 In a preferred embodiment, the first web page that appears offers an overview of 

the survey system, including its methodology of data collection, processing and 
aggregation. This page also offers the user a link to proceed directly to the test. When this 
option is chosen a new screen appears which displays test selection criteria with a sample 
Likert scale. At the bottom of this screen, the user is prompted to complete three empty 

1 5 fields for password protection entry. The first field is for the business ID 220 password that 
represents a unique identification of the organization. The second field is for job 
classification entry 230 that may include coding for employee classifications of executive 
staff 30, management staff 40, support staff 50 or direct customer contact staff 60. It may 
also include coding for external respondents, mentioned above, such as customers, 

20 suppliers, and the like, and there may be a different code, if desired, for each category of 
external respondents. The third field is for a unique employee ID 240 that relates to the 
preset numbers matching the number of employees as indicated above. 

Once this information is entered a confirmation process 250 is triggered that 
compares this keyed-in information to the information on server 70. If a match is detected 

25 the server then determines as at 260 if the user has previously initiated the survey. If the 
user has previously initiated the survey, a cache 280 record of the user's answers is 
accessed and the user is only presented with the survey questions that were not answered. 

If a mismatch 170 in passwords is detected during the login process210, an error 
screen is displayed and the user has the ability to reenter the three identification codes 220, 

30 230, and 240. It should be noted that additional fields may be added to the login process 
which may include length of employment, demographics and other data. Alternatively, a 
single login ID may be utilized which individually identifies the employee's organization, 
job classification and personal identification. 
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Upon successful login 210 and confirmation 250, the survey is commenced as at 
270 with a series of questions presented in HTML format. Under each question is a form 
input section coded in HTML which may comprise a check box, radio button, or edit box. 
Alternatively, other input means compatible with the user's web browser may be utilized 
5 including ActiveX or Java. Responses to the input means are stored in a databaselOO that 
may reside on the web server or at a remote location. The questions and answer input 
forms may be configured on a single scrollable page or on individual HTML pages joined 
by hyperlinks. 

FIG. 3 discloses a sample question screen as viewed through an Internet web 

1 0 browser comprising a plurality of input sections to each survey question topic. Users are 
prompted to respond to each of the questions in one or more components. In the following 
example, there are three components. Component A queries whether the respondent 
believes a specific performance criteria exists in their workplace. Component B queries the 
respondent to weigh their level of agreement or disagreement to a functional extrapolation 

15 of component A. Component C asks the respondent to check one of many methods how 
such a performance criteria could be improved. 

In the example of FIG. 3 two question components are presented. Question 1 
proposes a query on the perceived leadership skills in the company and provides for a 
Boolean response. The first query is followed up by another query that requests a 

20 quantified response with seven possible entries. In the preferred operation of the Internet 
system, none of the responses are selected when the user first downloads the page to their 
web browser. Each response must be completed for the "next question button" to become 
available for next question cueing and for the responses of that user to be filed in the 
database 100. Should the user attempt to skip over the questions without responding, the 

25 internal controls validate whether a response has been entered and repeat the same page to 
the user until a completed response is received by the web server. In a preferred 
embodiment, JavaScript is utilized to validate completed responses. The question-to- 
question responses are temporarily stored in a cache 250 until all survey questions are 
completed and the data is then committed to database 100 for analysis. An interrupt feature 

30 is available to all respondents who are unable to finish the survey in one sitting. If the 

respondent stops and logs off, he or she may restart the survey at any later time, by simply 
re-entering the password, job classification code and test ID. This procedure opens an 
incomplete survey and restores it to the last position or next available question from data 
stored in cache 250. When all questions are completed and a "save last answer question" 
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button is selected, the survey is flagged as done 260 for final database 100 entry. Analysis 
140 is available when all users complete their survey. Data from this part of the survey 
helps the organization evaluate the current level of organizational consistency about 
perceptions of effectiveness and efficiency between and among different levels of staff, 
5 across all operations, between staff members, and between and among the pool of external 
respondents as well, if any. As improvements are made the organization can revisit the 
assessment in future years to track scores in key areas, to monitor improvements and to 
encourage greater performance. A score card system also provides a context for an 
organization's scores to be compared to the respective scores of other organizations. 
10 In the touch-tone telephone embodiment shown in FIG 4 M the questions are audibly 

prerecorded for playback. The employee or non-employee is given a telephone number to 
call with a set of access codes which may comprise a business ID, job classification and 
employee ID, if applicable, as described in connection with FIG. 2. The user dials into the 
system from a standard touch-tone phone 300. The user is prompted for each identifier 

15 which is entered by pressing the numbers of the touch-tone telephone. When a key is 
pressed on a touch-tone telephone, a DTMF signal is emitted which is correlated to the 
numeral on the keypad by a system processor. These systems are widely used in automatic 
call distributor ("ACD") systems utilized by call centers and by banks and financial 
institutions to provide automated account information. In the example, after the user is 

20 authenticated prerecorded segments 320 of the survey are played back to the user. As an 
option, the user may press the STAR (*) or POUND (#) key on their telephone to repca 
the last playback segment. After a survey question is audibly presented to the user, the user 
then may answer the question depending on the response options using a DTMF input330. 
For a Boolean question which requires a "yes" or "no" response, the usermay be prompted 

25 to press "1" for "yes" or "2" for "no." Alternatively, should the survey question require a 
quantitative response, the user may be prompted to enter a range of numbers on the 
telephone keypad. For example, should the user "Almost Always Disagree" they would 
depress "L" If the user "Rarely Agrees," they would depress "2." The scale may continue 
so that should the user "Almost Always Agrees," they would depress "7." However, it 

30 should be noted that the telephonic entry is not limited to single digits. For example, 
should the user be required to indicate a percentage numeral of 55%, the entry may be 
executed by depressing in sequence: "5," "5," and then the "#" key. Should the user fail to 
respond to the system with a DTMF signal 330 within a predetermined period of time, the 
system may replay as at 340 the previous question proposed. 
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Both the Internet-based and telephonically-based system preferably time-stamp the 
entries wherein each time a response is posted to the web server or entered into the 
telephone system, a corresponding record of the date and time is also stored in the 
database. This may be utilized for an assessment of the amount of time needed on average 
5 to complete the evaluation and when each employee submitted their responses to the 
survey. 

FIG. 5 illustrates a diagrammatical overview of an organization's total quality 
output scorecard based on the Malcolm Baldrige criteria. However, other performance 
measurements and formulas may be applied within the scope of this invention. The 

10 purposes of the Baldrige Assessment are to promote an understanding of the requisites for 
performance achievement and competitiveness for business entities and to promote the 
sharing of information on successful business practices. The Baldrige Assessment consists 
of seven criteria: (1) leadership, (2) strategic planning, (3) customer and market focus, (4) 
information and analysis, (5) human resource development and management, (6) process 

1 5 management, and (7) business results. 

Leadership is evaluated by the ability of senior executives to guide the company 
and how the company addresses it responsibilities. Strategic planning examines how the 
company sets strategic directions and how it determines key action plans. Customer and 
market focus examines how the company determines the requirements and expectations of 

20 customers and markets. Information analysis examines the management, effective use, and 
analysis of data and information to support key company processes and the company's 
performance management system. Human resource development and management 
examines how the company enables its workforce to develop its full potential and how the 
workforce is aligned with the company' objectives. Process management examines aspects 

25 of how key production, delivery and support processes are designed, managed, and 

improved. Business results are examined by the company's performance and improvement 
in its key business areas: customer satisfaction, financial and marketplace performance, 
human resources, supplier and partner performance, and operation performance. Business 
results are also evaluated by the performance of the company relative to its competitors. 

30 In a preferred embodiment, each question has both a yes/no and weighted score 

component so that data can be accurately tallied with sophisticated analysis and report 
generation. Instead of using a simple 1-5 Likert scale which is subject to a wide range of 
interpretation, the current invention uses quantified behavioral descriptors, dramatically 
increasing score reliability. As previously illustrated in FIG. 3a written question is 
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followed by a quantified (percentage-tagged) behavioral description. This form of test 
construction yields an increase in the consistency and accuracy of ratings as well as greater 
coherence of linkages between business processes. The method of utilizing a written 
question followed by a quantified description is well known in the health care industry. In 
5 health care it was discovered that by obtaining input this way from a cross section of 
functions and levels throughout the organization, a performance profile can be developed 
that not only identifies strengths and areas for improvement, but deployment gaps as well. 
This is something that written narrative assessments alone do not effectively provide. 
Accurate survey data, based on behavioral follow-up questions, can be used to compare or 

10 benchmark organizations within and among industries, and can also support longitudinal 
performance studies. 

The system represents an item by item, group by group performance improvement 
digest that allows management the ability to recognize the linkage between performance of 
key processes to quality improvement outcomes (including the future allocation of 

1 5 resources to achieve such outcomes). 

The system does this by compiling each respondent's raw data and generating a 
written digest of an organization's strengths and opportunities for improvement. Data is 
gathered and assembled to provide an organization with a detailed analysis in the form of a 
printed report. Responses in the database 100 are analyzed as at 140 and reported in several 

20 ways. The database table is constructed to include vertical columns formatted with 
mathematical formulas representing yes or no response totals, raw Likert averages, 
adjusted Likert averages and Organizational Success Ratios ("OSRs"). Dividing the 
adjusted Likert by the raw Likert produces an OSR. This number can be represented in the 
form of a ratio or quotient and generally as a two place decimal ranging from .01 to a 

25 perfect 1 .00. The OSR is used to profile the system's highest and lowest scores in 

performance improvement, representing current quality strengths and weaknesses in the 
business entity. 

The OSR data collected from the employees and non-employees, if any, is 
consistent across the organization. The results can present a clear profile of strengths and 
30 current weaknesses from which the business entity can plan appropriate actions. Each OSR 
is then converted to a Baldrige equivalency formula that converts the two place decimal 
point to a percentage. The percentage number is then applied line by line to re-create the 
points available in each section found in the sample table. The "maximum" points for each 
category of the Malcolm Baldrige National Quality Award is listed with the actual score. 
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As illustrated in FIG. 5, a maximum of 450 points for the business results" 
category may be obtained, even if the sample organization returned only 160 points. To 
convert each business category result to a ratio, the total points achieved in each category 
is divided by the overall maximum number of points possible for the category. Thus, 160 
5 divided by 450 equals 0.356 or 35.6 percent. By converting each business category 

measurement to a ratio, the organization may easily obtain a quantifiable assessment of its 
performance. 

It will be seen that the objects set forth above, and those made apparent from the 
foregoing description, are efficiently attained. Since certain changes may be made in the 
10 above construction without departing from the scope of the invention, it is intended that al 
matters contained in the foregoing description or shown in the accompanying drawings 
shall be interpreted as illustrative and not in a limiting sense. 

It is also to be understood that the following claims are intended to cover all of the 
generic and specific features of the invention herein described, and all statements of the 
1 5 scope of the invention which, as a matter of language, might be said to fall therebetween. 
Now that the invention has been described, 
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WHAT IS CLAIMED IS: 

1. A method for quantifying predetermined criteria relating to the performance of 
an organization, comprising the steps of: 

5 an array of survey questions for evaluating predetermined performance criteria of 

an organization; 

a survey administrator; 

a data-gathering interface for a predetermined pool of respondents to individually 
record answers to said survey questions; 
10 a database accessible by said survey administrator containing answers submitted by 

said predetermined pool of respondents; and 

a predetermined formula for evaluating said answers; 

whereby members of said predetermined pool of respondents individually record 
answers to said survey questions through said data-gathering interface; 
15 whereby said survey administrator stores said answers in said database so that said 

predetermined formula produces a quantified result based on said predetermined criteria 
relating to the performance of said organization. 

2. The method of claim 1, wherein a dual tone multi-frequency telephone system is 
utilized for said data-gathering interface and wherein individual members of said 

20 predetermined pool of respondents dial into said telephone system maintained by said 

survey administrator, said telephone system audibly plays prerecorded questions based on 
said predetermined criteria, and said individual members are prompted to depress a 
particular key on a standard touch-tone telephone in response to said prerecorded questions 
which responses are recorded in said database. 

25 3. The method of claim 1 , wherein a remote web server is utilized to record said 

answers whereby said individual members of said predetermined pool of respondents 
access a website server on the Internet, said website containing said survey questions and 
an input means for said answers to be posted through said website server to said database. 

4. The method of claim 1, wherein said survey questions provide for a Boolean 
30 response followed by a quantified response to the same survey topic. 

5. The method of claim 1, wherein the Malcolm Baldrige National Quality Award 
Criteria as defined by the United States National Institute of Standards and Technology is 
utilized for said predetermined formula. 

6. The method of claim 1, wherein said quantified result is resolved by the steps of: 
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scoring the survey responses relative to a predetermined score based on 
determination on the importance of each performance criteria; 

correlating the survey responses for performance criteria categories into distinct 
arrays of data; 

5 adding numerical values to said arrays of data representing separate performance 

criteria categories; and 

dividing said numerical values by a maximum theoretical value of said 
performance criteria category. 

7. The method of claim 1, wherein said predetermined pool of respondents is an 
10 internal pool of respondents consisting of employees of the organization. 

8. The method of claim 1, wherein said predetermined pool of respondents is an 
external pool of respondents consisting of non-employees of the organization. 

9. A method for quantifying predetermined criteria relating to the performance of 
an organization, comprising the steps of: 

1 5 providing an array of survey questions for evaluating predetermined criteria of an 

organization; 

providing a survey administrator; 

defining a predetermined pool of respondents to whom said survey questions are to 
be propounded; 

20 using a remote web server to record answers whereby members of said 

predetermined pool of respondents access said web server on the Internet, one or more web 
pages present said survey questions and an input form for said answers to be posted 
through said web server to a database accessible by said survey administrator, and 
providing a predetermined formula for evaluating said answers based on the 

25 Malcolm Baldrige National Quality Award Criteria as defined by the United States 
National Institute of Standards and Technology; 

whereby said members of said predetermined pool of respondents record answers 
to said survey questions through said web server and said survey administrator stores said 
answers in said database wherein said predetermined formula produces a quantified result 

30 based on said predetermined criteria relating to the performance of said organization. 

10. The method of claim 9, wherein said members of said predetermined pool of 
respondents are employees of the organization. 

11. The method of claim 9, wherein said members of said predetermined pool of 
respondents are non-employees of said organization. 
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1 . Do senior leaders guide the company in setting direction and sustaining 
effective leadership through the organization? 

O Yes 
® No 



Senior leaders fully guide and sustain effective leadership. 

O Almost Always Disagree 
<§) Rarely Agree (20%) 
O Seldom Agree (35%) 
O Agree (50%) 
O Mostly Agree (65%) 
O Strongly Agree (80%) 
O Almost Always Agree 
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Scores for a Sample Organization 


Evaluation Categories 


Total Points 


Percent of 
Total Points 


Points Earned 


1. Leadership 


110 


38.2 


42 


2. Strategic Planning 


80 


43.8 


35 


3. Customer and Market Focus 


80 


40 


32 


4. Information and Analysis 


80 


25 


20 


5. Human Resource Focus 


100 


44 


44 


6. Process Management 


100 


36 


36 


7. Business Results 


450 


35.6 


160 


Totals 


1,000 


36.9 


369 
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SURVEY SYSTEM TO QUANTIFY VARIOUS CRITERIA RELATING TO THE OPERATION OF AN ORGANIZATION 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

5 The present invention relates generally to a survey system and apparatus for 

evaluating the results. More particularly, it relates to a remotely accessible system for the 
collection of employee or non-employee survey responses to quantify various criteria 
relating to the operation of an organization. 

2. Description of the Prior Art 

10 In an increasingly competitive marketplace, business entities strive to improve 

leadership, quality, customer satisfaction and other criteria that directly or indirectly relate 
to the ultimate profitability of the enterprise. Empirical evidence has established that 
business operations that excel in distinct and measurable attributes are far more likely to be 
profitable in commerce. As an example, the "Baldrige Index" is made up of publicly traded 

15 U.S. companies that have received the Malcolm Baldrige National Quality Award during 
the years 1988 to 1996. The Secretary of Commerce and the National Institute of Standards 
and Technology ("NIST") were given the responsibility, under Public Law 100-107, to 
develop and administer the Award with cooperation and financial support from the private 
sector. NIST "invested" a hypothetical $1,000 in each of the six whole company winners of 

20 the Baldrige Award. The investments were tracked from the first business day of the month 
following the announcement of award recipients (or the date they began public trading) to 
Dec. 1, 1997. Adjustments were made for stock splits. Another $1,000 was hypothetical ly 
invested in the Standard and Poor's Index 500 ("S&P") at the same time. NIST found that 
the group of six outperformed the S&P 500 by more than 2.7 to 1, achieving a 394.5 

25 percent return on investment compared to a 146.9 percent return for the S&P 500. 

Therefore, it is desirable for an organization to administer periodic assessments of its 
operations. 

The detailed assessment of a business operation is often a difficult, expensive, and 
time-consuming task. Typically, upper level management may knowingly or inadvertently 
30 affect the accurate measure of information gathered for the assessment. Outside consultants 
are often employed to interview and observe the operation of the enterprise on-site. 
However, management may wish to influence the data for a number of reasons. Upper 
level management may be greatly affected by the results of the evaluation and may attempt 
to direct the outside consultant only to well performing operations, shielding problem areas 
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from discovery. In addition, the outside consultant may interview subordinate employees 
in the presence of their supervisor This creates a poor environment for gaining candor 
from the subordinate employee on potential areas of improvement that are the 
responsibility of the supervisor. 
5 An evaluation by an outside consultant may require the consultant to travel to 

different geographical locations to assess a large operation with continuity. The consultant 
may also require the employee to stop productive work to allow time for the assessment 
interview. It is time-consuming for an independent consultant to conduct individual 
employee interviews, record the data, and assemble the information into a useful form. 
10 Furthermore, should a business wish to conduct periodic evaluations, there is no guarantee 
that the same consultant will be available. Therefore, the company cannot be assured that 
the next independent assessment performed will have the same consistency. Nor is there 
any assurance that secondary evaluations will produce meaningful results comparable to 
previous exercises. 

15 Should the company attempt to apply known economic principles to an "in-house" 

self-assessment, there is the potential that more harm than good may come from the 
endeavor. Many organizations begin the process of self-assessment with a shallow 
understanding of the performance criteria sought or the optimal method in which to gather 
the information. This self-assessment suffers from an inseparable relationship between the 

20 company's own evaluator and the upper management that may exert influence, not only on 
the examinees, but also on the examiner. 

Previous attempts have been made to provide business-related surveys such as 
described in U.S. Patent No. 5,551,880 to Bonnstetter et al (the 4 880 patent) which is 
incorporated herein by reference. The 4 880 patent describes a system for predicting the 

25 potential success of an individual for a particular job or task. A survey is conducted 

wherein the employee or potential employee submits information on behavioral and value 
preferences. The information is then analyzed and compared against standards for behavior 
and values previously resolved for specific employment. In a preferred embodiment of the 
'880 patent, the behavior and value questions are administered through software and 

30 evaluated via a predetermined algorithm. However, the system analyzes the individual 
employee and not the business entity as a whole. Nor does the *880 patent describe an 
independent party to administer the survey to ensure confidentially, honesty, and a full 
disclosure of the employee's perceptions of the business entity. 



2 



WO 00/41110 



PCT/US99/29432 



Therefore, there is a need in the art for a method and apparatus to provide a reliable 
self-assessment survey process that can be easily administered and scored with the 
accuracy and completeness of a well developed written narrative self-assessment. 

There is a further need in the art to integrate existing telecommunication 
5 technologies to avoid the time and labor intensive ordeal of the paper and pencil process or 
other traditional means of survey administration. This would allow the database of both 
individual responses and cumulative data to be completely external to the organization 
being assessed. 

There is a further need in the art to provide confidentiality to organizations and 
10 their employees participating in the survey. This provides a distinct opportunity to 

benchmark the data to industry peers and to provide the objective assurance that all data is 
accurate and verifiable by a reliable survey administrator organization. 

However, in view of the prior art in at the time the present invention was made, it 
was not obvious to those of ordinary skill in the pertinent art how the identified needs 
15 could be fulfilled. 

SUMMARY OF THE INVENTION 
The present invention solves significant problems in the art by providing a 
telecommunications infrastructure that supports the efficient, confidential and accurate 
20 measure of performance criteria relating to the performance of an organization. A 
predetermined set of performance criteria are measured by the use of an automated 
employee and non-employee interview system whereby recorded responses are stored in a 
database which is utilized for a quantitative evaluation of various aspects of a business 
enterprise. 

25 The present invention comprises an array of survey questions for evaluating a 

plurality of predetermined criteria relating to the performance of an organization. A survey 
administrator controls a data-gathering interface means for employees or non-employees of 
the business to confidentially record answers to the survey questions. For each individual 
employee survey, the data-gathering interface records the employee's identification, their 

30 job classification and a business identifier. Pertinent data is also collected for each non- 
employee participating in the survey. The data-gathering interface may comprise an 
Internet web server, a touch-tone telephone entry system, a paper and pencil system that 
compiles the answers by scanning them automatically, and other existing or to-be- 
developed interface means as well. For example, an additional data-gathering interface 
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means could be provided in the form of a personal computer having appropriate software. 
A respondent could save his or her survey responses on a diskette and mail the diskette to 
the survey administrator, or email the responses to the survey administrator as an attached 
file. This would enable a respondent to prepare answers and send them to the survey 
5 administrator while the respondent is on a flight, on the road, or otherwise away from a 
permanent office environment. 

The survey administrator maintains a database of the responses to the survey 
questions and has the ability to archive past response data. The survey administrator then 
applies a selected formula to the survey data to evaluate the performance of the 

10 organization and produces a quantifiable result based on a plurality of business categories. 

In a preferred embodiment, an organization wishing to evaluate the performance of 
various key aspects of its operations selects one or more business performance models. 
Each business performance model is measured using predetermined questions for 
employees of the business. A survey administrator performs the administration of the 

15 predetermined questions so that confidentially in the interview process insures candid 
answers to the questions. In one embodiment, responses to the questions are gathered 
utilizing an Internet web server and are stored either locally on the server or at a remote 
location for the survey administrator to later compile and analyze. The web server provides 
the opportunity to quickly and efficiently gather information, particularly where various 

20 offices of an organization are separated by substantial geographic distance. The web server 
also may operate 24 hours per day at relatively low cost and permit a plurality of 
employees to enter in responses at the same time. 

In an alternative embodiment, a telephone system utilizing dual-tone multi- 
frequency ("DTMF") input provides a means for employees to confidentially record 

25 information relating to the performance of a company. The DTMF entry has the advantage 
of not requiring the employee to have a machine capable of accessing the web server over 
the Internet. Using DTMF entry, the employee logs on using a code provided by the survey 
administrator. Once within the telephone system, pre-recorded questions are audibly 
played to the employee. At the conclusion of each question, the employee is prompted to 

30 record their response by depressing numerals on the telephone. 

In another embodiment, non-employees participate in the survey as well. However, 
the questions propounded to non-employees may differ from the questions propounded to 
employees. 
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The data is readily compiled into a useable database format. At the conclusion of 
the data gathering stage, the survey administrator can easily analyze the recorded 
information and produce a report to the business entity evaluating the entity based on the 
pre-selected business performance criteria desired. Furthermore, the operation of the 
5 invention is highly reproducible and consistent. A baseline assessment can be repeated 
over time, allowing the organization to track quantifiable results and improvements. 

It should be noted that the current invention has a wide range of applications which 
may include heath care accreditation, educational institution assessment, and similar tasks. 
For example, the current invention may be applied to a Performance-based Organizational 
10 Effectiveness & Efficiency Tracking Program ("POET") wherein the organizational 
effectiveness and efficiency of child welfare organizations are measured in addition to 
performance indicators essential to success in a managed care environment. 

An advantage of the invention is that a complete assessment of a large corporation 
may be completed more rapidly and efficiently than using independent consultants that 
15 interview onsite. The reduced costs and speed permit a business to conduct more frequent 
performance evaluations that can be utilized to operate the business more profitably. 

Another advantage of the invention is that employees of the business are provided 
with the opportunity to submit honest and candid answers to potentially sensitive 
questions. Because the opportunity for external influence to affect the employee's response 
20 is curtailed, the final evaluation of the business entity is more likely to uncover problem 
areas in its operations that may be subsequently improved. 

Another advantage of the invention is that the infrastructure of the evaluation is 
highly consistent over time. This permits the business entity to periodically measure itself 
against previous assessments. 
25 These and other important objects, advantages, and features of the invention will 

become clear as this description proceeds. 

The invention accordingly comprises the features of construction, combination of 
elements, and arrangement of parts that will be exemplified in the description set forth 
hereinafter and the scope of the invention will be indicated in the claims. 

30 

BRIEF DESCRIPTION OF THE DRAWINGS 
For a fuller understanding of the nature and objects of the invention, reference 
should be made to the following detailed description, taken in connection with the 
accompanying drawings, in which: 
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FIG. 1 is a diagrammatical view of the survey process according to the invention; 

FIG. 2 is a diagrammatical view of an Internet-based method of retrieving and 
recording employee responses; 

FIG. 3 is an illustrative example of a survey question presented to a user from an 
5 Internet web server; 

FIG. 4 is a diagrammatical view of a telephone-based method of retrieving and 
recording employee responses; and 

FIG. 5 is an illustrative example of an organization's performance scorecard based 
on the Malcolm Baldrige criteria. 

10 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
Referring initially to FIG. 1, it will there be seen that the organization subject to 
the survey is denoted by the reference number 10 as a whole. Virtually all business entities 
have the following job classifications: executive staff 30, management staff 40, support 

1 5 staff 50, and customer contact staff 60 as integral employee components which support 
most all business functions of the enterprise. It is each of these job classifications of 
employees who form the internal pool of respondents of the business entities. The login 
procedures for these internal respondents will be discussed in more detail later. 

In another embodiment of the invention, an external pool of respondents is also 

20 included. The external pool of respondents may include consumers, customers or clients of 
the business entity, i.e. y those who are not employees or otherwise a part of the internal 
group of respondents. The external pool of respondents may also be adjusted as desired to 
include members of the board of directors or trustees, suppliers or vendors, people who 
perform outsourced operations, competitors, partners, consultants, or any other relevant 

25 external respondent entity. The login procedure for these external respondents is similar to 
the login procedure for internal respondents, except that said login procedure identifies all 
members of the external pool of respondents as such and by their specific category such as 
customer, vendor, or the like. External respondents may also be queried as to their age, 
frequency of contacts with the business entity, and other information deemed pertinent. 

30 In many organizations there are one or more consultants or holders of other job 

titles who are on the business premises daily and whom might seem to be employees of the 
organization even though they are technically independent contractors. There are other jobs 
as well where the distinction between employee or non-employee may not be easily 
ascertainable. This invention does not depend upon any legal analysis of who should be 
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considered internal or external to the organization as a matter of legal definition because in 
some cases a non-employee such as an outside consultant might best be included as a 
member of the internal pool of respondents for survey purposes. Therefore, for 
convenience purposes, and not in any strict legal sense, the members of the pool of internal 
5 respondents are herein deemed employees of the organization, and the members of the pool 
of external respondents are herein deemed non-employees of the organization. A survey 
administrator is free to determine the respective members of the pools of internal and 
external respondents in any way that best serves the purposes of the survey without regard 
to strict legal determinations of the legal status of the individual members of either pool. 

1 0 The system obtains information from individual employees or other respondents 

regarding their perception of performance criteria existing in the workplace. This is 
accomplished by utilizing a performance criteria survey. This survey is administered and 
controlled by a survey administrator denoted 20. The survey data is obtained by a data- 
gathering interface 70. The data-gathering interface may comprise a variety of 

1 5 telecommunication mediums which may include touch-tone telephone systems, hypertext 
language format ("HTML") Internet web servers, client-server wide area networks 
("WANs") or local area networks ("LANs"). Data-gathering interface 70 acts as a conduit 
to receive survey data 90 in response to a plurality of predetermined survey questions. 
Each question contains several important aspects of a business function linked to vital 

20 performance abilities of the organization. 

The results of the survey are stored in a database 100 for a baseline analysis of 
performance criteria and may also be used longitudinally in later intervals of time. Prior 
survey data from prior years 110 and 120 may be archived by the survey administrator 20 
and later utilized as a base-line reference to the current evaluation of the organization. 

25 Once a requisite number of respondents or all respondents complete surveys, an analysis 
140 is performed on the survey data 90 by the survey administrator 20 and then 
communicated back to the organization 10. 

FIG. 2 provides an overview of an Internet-based data and tracking system. A 
remote user terminal 160 accesses an Internet web server 70 through a public switched 

30 network ("PSN") 190. The survey administrator 20 provides the organization 10 with a 
series of codes for the login 2 10 and test administration to begin. Typically, the 
organization 10 will choose an alphanumeric password of ten digits or less for 
identification of the organization 10 as a whole. The survey administrator20 records said 
password and opens it for usage in protected area of the website. Next the survey 
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administrator 20 assigns a chronological range of numbers sufficient to match the total 
number of respondent employees or non-employees elected and eligible to sit for the 
survey. To begin the login 210 process, the respondent accesses the world wide web and 
web server 70 using its uniform resource locator ("URL"). The URL may be provided as a 
5 numerical-based Internet protocol ("IP") address or may be accessed by a registered 
domain name via a domain name server ("DNS"). 

It should also be noted that alternatives to web-based Internet communications may 
be employed. The alternatives may include 3270 emulation, VT-100, ASCII or separate 
client-server applications. 

10 In a preferred embodiment, the first web page that appears offers an overview of 

the survey system, including its methodology of data collection, processing and 
aggregation. This page also offers the user a link to proceed directly to the test. When this 
option is chosen a new screen appears which displays test selection criteria with a sample 
Likert scale. At the bottom of this screen, the user is prompted to complete three empty 

15 fields for password protection entry. The first field is for the business ID 220 password that 
represents a unique identification of the organization. The second field is for job 
classification entry 230 that may include coding for employee classifications of executive 
staff 30, management staff 40, support staff 50 or direct customer contact staff 60. It may 
also include coding for external respondents, mentioned above, such as customers, 

20 suppliers, and the like, and there may be a different code, if desired, for each category of 
external respondents. The third field is for a unique employee ID 240 that relates to the 
preset numbers matching the number of employees as indicated above. 

Once this information is entered a confirmation process 250 is triggered that 
compares this keyed-in information to the information on server 70. If a match is detected 

25 the server then determine^as at 260 if the user has previously initiated the survey. If the 
user has previously initiated the survey, a cache 280 record of the user's answers is 
accessed and the user is only presented with the survey questions that were not answered. 

If a mismatch 170 in passwords is detected during the login process210, an error 
screen is displayed and the user has the ability to reenter the three identification codes 220, 

30 230, and 240. It should be noted that additional fields may be added to the login process 
which may include length of employment, demographics and other data. Alternatively, a 
single login ID may be utilized which individually identifies the employee's organization, 
job classification and personal identification. 
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Upon successful login 210 and confirmation 250, the survey is commenced as at 
270 with a series of questions presented in HTML format Under each question is a form 
input section coded in HTML which may comprise a check box, radio button, or edit box. 
Alternatively, other input means compatible with the user's web browser may be utilized 
5 including ActiveX or Java. Responses to the input means are stored in a databaselOO that 
may reside on the web server or at a remote location. The questions and answer input 
forms may be configured on a single scrollable page or on individual HTML pages joined 
by hyperlinks. 

FIG. 3 discloses a sample question screen as viewed through an Internet web 

10 browser comprising a plurality of input sections to each survey question topic. Users are 
prompted to respond to each of the questions in one or more components. In the following 
example, there are three components. Component A queries whether the respondent 
believes a specific performance criteria exists in their workplace. Component B queries the 
respondent to weigh their level of agreement or disagreement to a functional extrapolation 

15 of component A. Component C asks the respondent to check one of many methods how 
such a performance criteria could be improved. 

In the example of FIG. 3 two question components are presented. Question I 
proposes a query on the perceived leadership skills in the company and provides for a 
Boolean response. The first queiy is followed up by another query that requests a 

20 quantified response with seven possible entries. In the preferred operation of the Internet 
system, none of the responses are selected when the user first downloads the page to their 
web browser. Each response must be completed for the "next question button" to become 
available for next question cueing and for the responses of that user to be filed in the 
database 100. Should the user attempt to skip over the questions without responding, the 

25 internal controls validate whether a response has been entered and repeat the same page to 
the user until a completed response is received by the web server. In a preferred 
embodiment, JavaScript is utilized to validate completed responses. The question-to- 
question responses are temporarily stored in a cache 250 until all survey questions are 
completed and the data is then committed to database 100 for analysis. An interrupt feature 

30 is available to all respondents who are unable to finish the survey in one sitting. If the 

respondent stops and logs off, he or she may restart the survey at any later time, by simply 
re-entering the password, job classification code and test ID. This procedure opens an 
incomplete survey and restores it to the last position or next available question from data 
stored in cache 250. When all questions are completed and a "save last answer question" 
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button is selected, the survey is flagged as done 260 for final database 100 entry. Analysis 
140 is available when all users complete their survey. Data from this part of the survey 
helps the organization evaluate the current level of organizational consistency about 
perceptions of effectiveness and efficiency between and among different levels of staff, 
5 across all operations, between staff members, and between and among the pool of external 
respondents as well, if any. As improvements are made the organization can re-visit the 
assessment in future years to track scores in key areas, to monitor improvements and to 
encourage greater performance. A score card system also provides a context for an 
organization's scores to be compared to the respective scores of other organizations. 

10 In the touch-tone telephone embodiment shown in FIG 4., the questions are audibly 

prerecorded for playback. The employee or non-employee is given a telephone number to 
call with a set of access codes which may comprise a business ID, job classification and 
employee ID, if applicable, as described in connection with FIG, 2. The user dials into the 
system from a standard touch-tone phone 300. The user is prompted for each identifier 

15 which is entered by pressing the numbers of the touch-tone telephone. When a key is 
pressed on a touch-tone telephone, a DTMF signal is emitted which is correlated to the 
numeral on the keypad by a system processor. These systems are widely used in automatic 
call distributor ("ACD") systems utilized by call centers and by banks and financial 
institutions to provide automated account information. In the example, after the user is 

20 authenticated prerecorded segments 320 of the survey are played back to the user. As an 
option, the user may press the STAR (*) or POUND (#) key on their telephone to repel 
the last playback segment. After a survey question is audibly presented to the user, the user 
then may answer the question depending on the response options using a DTMF input330. 
For a Boolean question which requires a "yes" or "no" response, the usermay be prompted 

25 to press "1" for "yes" or "2" for "no." Alternatively, should the survey question require a 
quantitative response, the user may be prompted to enter a range of numbers on the 
telephone keypad. For example, should the user "Almost Always Diagree," they would 
depress "1." If the user "Rarely Agrees," they would depress "2." The scale may continue 
so that should the user "Almost Always Agrees," they would depress "7." However, it 

30 should be noted that the telephonic entry is not limited to single digits. For example, 
should the user be required to indicate a percentage numeral of 55%, the entry may be 
executed by depressing in sequence: "5," "5," and then the "#" key. Should the user fail to 
respond to the system with a DTMF signal 330 within a predetermined period of time, the 
system may replay as at 340 the previous question proposed. 
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Both the Internet-based and telephonically-based system preferably time-stamp the 
entries wherein each time a response is posted to the web server or entered into the 
telephone system, a corresponding record of the date and time is also stored in the 
database. This may be utilized for an assessment of the amount of time needed on average 
5 to complete the evaluation and when each employee submitted their responses to the 
survey. 

FIG. 5 illustrates a diagrammatical overview of an organization's total quality 
output scorecard based on the Malcolm Baldrige criteria. However, other performance 
measurements and formulas may be applied within the scope of this invention. The 

10 purposes of the Baldrige Assessment are to promote an understanding of the requisites for 
performance achievement and competitiveness for business entities and to promote the 
sharing of information on successful business practices. The Baldrige Assessment consists 
of seven criteria: (I) leadership, (2) strategic planning, (3) customer and market focus, (4) 
information and analysis, (5) human resource development and management, (6) process 

1 5 management, and (7) business results. 

Leadership is evaluated by the ability of senior executives to guide the company 
and how the company addresses it responsibilities. Strategic planning examines how the 
company sets strategic directions and how it determines key action plans. Customer and 
market focus examines how the company determines the requirements and expectations of 

20 customers and markets. Information analysis examines the management, effective use, and 
analysis of data and information to support key company processes and the company's 
performance management system. Human resource development and management 
examines how the company enables its workforce to develop its full potential and how the 
workforce is aligned with the company' objectives. Process management examines aspects 

25 of how key production, delivery and support processes are designed, managed, and 

improved. Business results are examined by the company's performance and improvement 
in its key business areas: customer satisfaction, financial and marketplace performance, 
human resources, supplier and partner performance, and operation performance. Business 
results are also evaluated by the performance of the company relative to its competitors. 

30 In a preferred embodiment, each question has both a yes/no and weighted score 

component so that data can be accurately tallied with sophisticated analysis and report 
generation. Instead of using a simple 1-5 Likert scale which is subject to a wide range of 
interpretation, the current invention uses quantified behavioral descriptors, dramatically 
increasing score reliability. As previously illustrated in FIG- 3 a written question is 
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followed by a quantified (percentage-tagged) behavioral description. This form of test 
construction yields an increase in the consistency and accuracy of ratings as well as greater 
coherence of linkages between business processes. The method of utilizing a written 
question followed by a quantified description is well known in the health care industry. In 
5 health care it was discovered that by obtaining input this way from a cross section of 
functions and levels throughout the organization, a performance profile can be developed 
that not only identifies strengths and areas for improvement, but deployment gaps as well. 
This is something that written narrative assessments alone do not effectively provide. 
Accurate survey data, based on behavioral follow-up questions, can be used to compare or 

10 benchmark organizations within and among industries, and can also support longitudinal 
performance studies. 

The system represents an item by item, group by group performance improvement 
digest that allows management the ability to recognize the linkage between performance of 
key processes to quality improvement outcomes (including the future allocation of 

1 5 resources to achieve such outcomes). 

The system does this by compiling each respondent's raw data and generating a 
written digest of an organization's strengths and opportunities for improvement. Data is 
gathered and assembled to provide an organization with a detailed analysis in the form of a 
printed report. Responses in the database 100 are analyzed as at 140 and reported in several 

20 ways. The database table is constructed to include vertical columns formatted with 
mathematical formulas representing yes or no response totals, raw Likert averages, 
adjusted Likert averages and Organizational Success Ratios ("OSRs"). Dividing the 
adjusted Likert by the raw Likert produces an OSR. This number can be represented in the 
form of a ratio or quotient and generally as a two place decimal ranging from .0 1 to a 

25 perfect LOO. The OSR is used to profile the system's highest and lowest scores in 

performance improvement, representing current quality strengths and weaknesses in the 
business entity. 

The OSR data collected from the employees and non-employees, if any, is 
consistent across the organization. The results can present a clear profile of strengths and 
30 current weaknesses from which the business entity can plan appropriate actions. Each OSR 
is then converted to a Baldrige equivalency formula that converts the two place decimal 
point to a percentage. The percentage number is then applied line by line to re-create the 
points available in each section found in the sample table. The "maximum" points for each 
category of the Malcolm Baldrige National Quality Award is listed with the actual score. 
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As illustrated in FIG. 5, a maximum of 450 points for the "business results" 
category may be obtained, even if the sample organization returned only 160 points. To 
convert each business category result to a ratio, the total points achieved in each category 
is divided by the overall maximum number of points possible for the category. Thus, 160 
5 divided by 450 equals 0.356 or 35.6 percent. By converting each business category 

measurement to a ratio, the organization may easily obtain a quantifiable assessment of its 
performance. 

It will be seen that the objects set forth above, and those made apparent from the 
foregoing description, are efficiently attained. Since certain changes may be made in the 
10 above construction without departing from the scope of the invention, it is intended that all 
matters contained in the foregoing description or shown in the accompanying drawings 
shall be interpreted as illustrative and not in a limiting sense. 

It is also to be understood that the following claims are intended to cover all of the 
generic and specific features of the invention herein described, and all statements of the 
1 5 scope of the invention which, as a matter of language, might be said to fall therebetween. 
Now that the invention has been described, 
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WHAT IS CLAIMED IS: 

1, A method for quantifying predetermined criteria relating to the performance of 
an organization, comprising the steps of: 

5 an array of survey questions for evaluating predetermined performance criteria of 

an organization; 

a survey administrator; 

a data-gathering interface for a predetermined pool of respondents to individually 
record answers to said survey questions; 
10 a database accessible by said survey administrator containing answers submitted by 

said predetermined pool of respondents; and 

a predetermined formula for evaluating said answers; 

whereby members of said predetermined pool of respondents individually record 
answers to said survey questions through said data-gathering interface; 
15 whereby said survey administrator stores said answers in said database so that said 

predetermined formula produces a quantified result based on said predetermined criteria 
relating to the performance of said organization. 

2. The method of claim 1, wherein a dual tone multi-frequency telephone system is 
utilized for said data-gathering interface and wherein individual members of said 

20 predetermined pool of respondents dial into said telephone system maintained by said 

survey administrator, said telephone system audibly plays prerecorded questions based on 
said predetermined criteria, and said individual members are prompted to depress a 
particular key on a standard touch-tone telephone in response to said prerecorded questions 
which responses are recorded in said database. 

25 3. The method of claim 1, wherein a remote web server is utilized to record said 

answers whereby said individual members of said predetermined pool of respondents 
access a website server on the Internet, said website containing said survey questions and 
an input means for said answers to be posted through said website server to said database. 

4. The method of claim 1, wherein said survey questions provide for a Boolean 
30 response followed by a quantified response to the same survey topic. 

5. The method of claim 1, wherein the Malcolm Baldrige National Quality Award 
Criteria as defined by the United States National Institute of Standards and Technology is 
utilized for said predetermined formula. 

6. The method of claim 1, wherein said quantified result is resolved by the steps of: 
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scoring the survey responses relative to a predetermined score based on 
determination on the importance of each performance criteria; 

correlating the survey responses for performance criteria categories into distinct 
arrays of data; 

5 adding numerical values to said arrays of data representing separate performance 

criteria categories; and 

dividing said numerical values by a maximum theoretical value of said 
performance criteria category. 

7. The method of claim 1, wherein said predetermined pool of respondents is an 
10 internal pool of respondents consisting of employees of the organization. 

8. The method of claim 1, wherein said predetermined pool of respondents is an 
external pool of respondents consisting of non-employees of the organization. 

9. A method for quantifying predetermined criteria relating to the performance of 
an organization, comprising the steps of: 

15 providing an array of survey questions for evaluating predetermined criteria of an 

organization; 

providing a survey administrator; 

defining a predetermined pool of respondents to whom said survey questions are to 
be propounded; 

20 using a remote web server to record answers whereby members of said 

predetermined pool of respondents access said web server on the Internet, one or more web 
pages present said survey questions and an input form for said answers to be posted 
through said web server to a database accessible by said survey administrator, and 
providing a predetermined formula for evaluating said answers based on the 

25 Malcolm Baldrige National Quality Award Criteria as defined by the United States 
National Institute of Standards and Technology; 

whereby said members of said predetermined pool of respondents record answers 
to said survey questions through said web server and said survey administrator stores said 
answers in said database wherein said predetermined formula produces a quantified result 

30 based on said predetermined criteria relating to the performance of said organization. 

10. The method of claim 9, wherein said members of said predetermined pool of 
respondents are employees of the organization. 

1 1. The method of claim 9, wherein said members of said predetermined pool of 
respondents are non-employees of said organization. 
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1 . Do senior leaders guide the company in setting direction and sustaining 
effective leadership through the organization? 

O Yes 
• NO 



Senior leaders fully guide and sustain effective leadership. 

O Almost Always Disagree 
<§> Rarely Agree (20%) 
O Seldom Agree (35%) 
O Agree (50%) 
O Mostly Agree (65%) 
O Strongly Agree (80%) 
O Almost Always Agree 
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Scores for a Sample Organization 




Evaluation Categories 


Total Points 


Percent of 
Total Points 


Points Earned 


1. Leadership 


no 


38.2 


42 


2. Strategic Planning 


80 


43.8 


35 


3. Customer and Market Focus 


80 


40 


32 


4. Information and Analysis 


80 


25 


20 


5. Human Resource Focus 


100 


44 


44 


6. Process Management 


too 


36 


36 


7. Business Results 


450 


35.6 


160 


Totals 


1,000 


36.9 


369 
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